fix(activity): reconcile completed tracker job status in history requests - #773

#25)

## What changed

PR numbers like lklynet#842 below refer to upstream `lklynet/aurral` pull
requests.

Brings the fork up to date with the parts of upstream `lklynet/aurral`
(2.9.0 → 2.10.0 and later, 102 commits since the fork point `7a42f9a6`)
that are worth having on top of the Postgres data layer. Upstream is not
merged wholesale: a trial merge produced 98 conflicted files because the
fork replaced SQLite, so every change was cherry-picked or hand-ported
with `(cherry picked from commit …)` / `Partial port of commit …`
references and adapted to the async `db.*` helpers.

**Security and auth**
- lklynet#792 `AUTH_PROXY_ENABLED=false` now wins over a configured proxy
header (the value is matched case-insensitively).
- lklynet#795 / lklynet#819 API-key-only requests can no longer create ownerless flows
or write user-owned data (favorites, play events,
Last.fm/ListenBrainz/Koito linking); they get 400/403. The same guard
now also covers playlist creation, import, the
Spotify/ListenBrainz/Last.fm imports and discover adoption.
- lklynet#870 Subsonic token auth works for every local user and follows
password changes. Credentials are stored encrypted with the settings key
in the new `users.subsonic_password` column (migration
`0005_users_subsonic_password`). Token auth needs the plaintext password
(MD5(password + salt)), so this is reversible encryption, the same
trade-off Navidrome makes.
- lklynet#613 identity-based OIDC, Google and Plex login and user lifecycle
(active/suspended/disabled, protected recovery admin, linked accounts,
15-minute re-auth for sensitive changes). Migration
`0006_user_identities` adds the tables/columns and runs upstream's
one-time backfill. Postgres adaptations: row locks for account adoption
and identity removal, 23505 mapped to 409, an in-memory mirror of
inactive owners so the download worker never queries per job. See the
commit message of c76db7b for the full list.

**Features**
- lklynet#741 reuse matching files already on disk before downloading.
- lklynet#846 per-playlist track availability with retry-missing.
- lklynet#883 listening-history toggle for flows and static playlists (More
menu).
- lklynet#889 webhook test button.
- lklynet#861 (+ lklynet#872, lklynet#876, lklynet#903) beets-backed matching engine for all
download sources, plus lklynet#899 album-only search fallback. The image gains
a Python venv with beets 2.14.1 at `/opt/aurral-matcher`, precompiled so
each match call starts warm. `/api/health` reports the matcher status,
and the Preview smoke test waits for its startup self-test.

**Fixes by area**
- Lidarr/library: lklynet#850 album requests stay monitored, lklynet#810 dedupe retry
jobs, lklynet#821/lklynet#854 Aurral ID markers read from native ID3 comments and
written to the grouping tag (no longer shown as Navidrome descriptions),
lklynet#848 Aurral-only artists stay out of Lidarr membership, lklynet#789 no library
scan per completed flow track, lklynet#845 (partial) every existing custom
format is listed in the Aurral profile, lklynet#799 (partial) structured log
for no-response Lidarr calls.
- Downloads/metadata: lklynet#793 yt-dlp channel validation, lklynet#890 yt-dlp node
runtime, lklynet#797 deemix reuses existing files, lklynet#852 slskd without API key,
lklynet#869 slskd missing download roots, lklynet#877 and lklynet#915 BrainzMash
caching/backoff and linked metadata first, lklynet#796 axios honours proxy env
vars.
- Playlists/activity: lklynet#893 large Spotify playlists (current `item`
wrapper, completeness check, no partial overwrite), lklynet#773 reused
downloads reconcile in Activity, lklynet#794 task queue counts beyond 500 rows.
- Media servers: Navidrome lklynet#774/lklynet#867/lklynet#887, Jellyfin lklynet#833 (settings save
no longer waits for library enumeration), Subsonic lklynet#791/lklynet#837.
- Discovery/other: lklynet#764, lklynet#778, lklynet#800, lklynet#798, lklynet#856, lklynet#808, lklynet#882 (jemalloc
decay, lower sharp/artwork concurrency).

**From lklynet#874 without the process split.** Background jobs stay
in-process: the fork's worker-thread scans and async Postgres already
remove the main-thread stalls lklynet#874 targets, and 11 processes would share
one Honker SQLite file and up to ~130 Postgres connections. Taken
instead:
- per-scope flow operation tokens (fixes a read-modify-write race);
- a hardened playlist mutation guard (partial-block rollback, release
always unblocks and prunes);
- `AURRAL_LIBRARY_SCAN_TIMEOUT_MS` (default 6 h) so a hung scan thread
is stopped instead of blocking every later scan.

**Playback-file retention (lklynet#842).** Automatic cleanup (flow refresh,
import sync, quality upgrade, fallback cleanup, download-folder
migration) no longer deletes files that a Plex, Navidrome or Jellyfin
playlist still references. Referenced files stay in place and are
retried on later library scans. If a service can't be checked, deletion
is deferred. Plex checks the server owner and every linked account,
using the fork's token-reconnect recovery. Navidrome needs an admin
account to see private playlists. Deliberate deletes and resets are
unchanged.

**Fixes from the final review** (several are upstream bugs too)
- `X-Forwarded-For` could spoof trust: with proxy auth on, a client
reaching Aurral directly could claim the proxy's address and sign in as
any user, even with `AUTH_PROXY_TRUSTED_IPS` set; with the local-network
bypass on, claiming `127.0.0.1` gave the sole admin. The allowlist now
checks the connecting address only, and the bypass needs every address
in the chain to be local.
- Any signed-in user could read or rotate the instance API key (which
authenticates as admin); both routes are now admin-only.
- An admin password reset now ends the account's sessions, websockets
and stream tokens. The protected recovery account can no longer be
demoted or deleted. Google login/exchange and Plex PIN creation share
the login rate limit.
- Stream and artwork routes served files without credentials on installs
with user accounts (or OIDC) but no `AUTH_PASSWORD`. They now follow the
same rule as the rest of the API.
- `updateUser` rewrote the whole row from an unlocked read (lost
updates); it now locks the row. Password logins go through
`recordPasswordLogin`, which only writes while the row still holds the
verified hash, so a login racing a password change cannot restore the
old password.
- JSON-backed stores (Plex/Navidrome/Jellyfin playlist pointers,
Plex/Spotify/scrobble connections) lost concurrent updates; writes are
now serialized. A Spotify token refresh can no longer resurrect a
cleared connection.
- lklynet#842 retention: an approved deletion clears the file's old retention
record, and Plex token rotation or sync errors no longer mark a cleanup
batch "usage unknown".
- Matching (lklynet#861): yt-dlp files were named by video id without tags, so
every yt-dlp download was rejected after downloading; titles like "Song
- Remastered 2009" were rejected or sent to review; "Live Forever"-style
titles were rejected as live versions on Soulseek/yt-dlp.
- lklynet#797: a deemix quality upgrade to the same path deleted the new
download and recorded the new tier on the old file.
- lklynet#741: on-disk reuse missed folders starting with "." ("...And Justice
for All").
- Flow record-history toggle in the flow menu called the shared-playlist
endpoint (upstream bug).
- `AURRAL_LIBRARY_SCAN_TIMEOUT_MS` above ~24.8 days made every scan time
out immediately; it is now capped.
- Pre-existing fork bugs found on the way: flow plans ignored the
owner's listening history (un-awaited profile), retry-registry writes
raced, disabling one news feed wiped all feeds and two news routes
answered `{}`.

**Fork-side fixes found while porting**
- Upstream's discovery refresh-loop fix (lklynet#764) read `.length` off a
promise on this fork, which silently disabled the missing-genre retry.
`discoveryNeedsRefresh` is now async and awaited.
- The lklynet#846 track-availability route returned before the async playlist
update finished. It now awaits it, as does the new lklynet#883 route.
- Upstream tests brought in by the picks were moved to the Postgres
harness. Upstream-only Playwright specs were dropped (the fork has no
e2e runner).

## Why

The fork was about 100 commits behind upstream, including security fixes
and features worth having. Porting commit by commit keeps the Postgres
conversion intact.

## Not ported (deliberately)

- **Jellyfin sync stack (lklynet#765/lklynet#812/lklynet#826)**: large; only needed with
Jellyfin.
- **lklynet#895 cancel downloads before playlist removal**: needs a redesign of
the download tracker on Postgres.
- **Library ownership chain (lklynet#756/lklynet#855/lklynet#917) and lklynet#768 available-only
default**: lklynet#768 is a product decision (it defaults to ON and hides
undownloaded albums).
- **lklynet#780 OpenSubsonic, lklynet#898 Navidrome stale-pointer recovery, lklynet#838
Navidrome prefix toggle, lklynet#835 unchanged-file scan skip, lklynet#790 per-artist
album fetch, lklynet#888, lklynet#884 logging, lklynet#900 UI lint, lklynet#817 Playwright**
- **SQLite-only changes** (lklynet#912, lklynet#824, lklynet#913 test tooling), the lklynet#874
process split, and upstream dependabot bumps (the fork's own dependabot
keeps its lockfile current).

## Known gaps

- The OIDC/SSO security review was done after the first ready-for-review
push (fixes above). Reviewed and fine: the OIDC flow (PKCE, state,
nonce, cookie-bound transaction and exchange code, status re-check),
Google and Plex link-only logins keyed by provider subject, session
lookup status checks, websocket auth, re-auth scoping, identity unlink
locking and migration 0006. Still open by design: with proxy auth
enabled and no `AUTH_PROXY_TRUSTED_IPS`, any client that can reach
Aurral may send the identity header (documented as required config).
- Reviewer notes left as-is: dead exports in
`weeklyFlowYtdlpSearch.js`/`weeklyFlowSoulseekSearch.js` (kept identical
to upstream); README and `docs/getting-started/docker.mdx` still point
at `ghcr.io/lklynet/aurral` rather than this fork's image; `lastfm.mdx`
still says an admin connects each user's Last.fm account.
- Docs were not built locally (no `docs/node_modules`); CI's docs build
covers it.

## Scope checklist

- [ ] This pull request has one clear purpose. Its purpose is syncing
upstream, but it bundles many upstream PRs.
- [ ] I kept unrelated fixes, refactors, formatting changes, dependency
updates, and features out of this pull request. It includes a few
fork-side fixes that the ports exposed (listed above).
- [ ] If this adds a feature, I linked the approved feature request or
included the Discord context in the Why section. Not applicable: these
are upstream features.

## Linked issue

None.

## Testing

Local environment: Node 26.8.2, PostgreSQL 18.6, ffmpeg 6.1.

- Baseline (`main`, 307c7dd): lint clean, unit 985/985, integration
57/57.
- Full run on 968385a (after the lklynet#613 OIDC port): lint clean, unit
1256/1256, integration 90/90, frontend build OK. CI green on that head.
- Full run on 5714d20: lint clean, unit 1271/1271, integration 90/90,
frontend build OK. The security-review commits after it (2e189c5,
d9adcd8) pass lint and the auth and user suites (unit 82/82,
integration 64/64; the two auth files again at 26/26 after the spoofing
tests were added). Every new test fails without its fix.
- Most review fixes add a regression test that fails without the fix and
passes with it. The exceptions are covered only by the existing suites:
the flow-menu history toggle, the deemix upgrade reuse, the
dotted-folder lookup, and the un-awaited history/registry writes.
- Not run: Docker image build and the Playwright smoke tests (no Docker
daemon or e2e harness here). CI's Preview workflow builds the image and
runs the matcher self-test.

## Release impact

- [ ] Major: incompatible change
- [x] Minor: backward-compatible feature
- [ ] Patch: backward-compatible fix
- [ ] None: documentation, CI, tests, or internal-only change

Database migrations: `0005_users_subsonic_password` (additive column)
and `0006_user_identities` (lklynet#613). They are applied at startup under the
existing advisory lock.

Upgrade notes (lklynet#613):
- Every session expires once; all users sign in again.
- OIDC users are now matched by issuer and subject. Before telling users
the upgrade is live, an admin should approve adoption of each existing
OIDC account (Settings > Users). Otherwise the first SSO sign-in creates
a new `-2` account.
- Subsonic token-only clients need the user to sign in to Aurral once
(or an admin password reset) before they work (lklynet#870).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_011txmpoBa1acyqWaPZUzyRo

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Google and Plex sign-in, linked-account management, SSO-only
sign-in, and account status controls.
* Added per-playlist listening-history and track-availability settings,
with availability indicators and re-search actions.
* Added webhook testing and optional country codes for nearby-show
searches.
* **Improvements**
* Improved track matching and download review across sources, and
protects files still referenced by connected media-server playlists
during automatic cleanup.
* Supports slskd configurations without an API key and complete Spotify
playlist imports.
* Improved metadata lookups, playlist publishing, and account security.
* Added Navidrome path mappings for cleanup checks and public visibility
for newly created Navidrome playlists.
  * Prefers metadata-provider genres when available.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Barbaros G. <tayyipgoren@gmail.com>
Co-authored-by: Lee Kelly <hello@leekelly.org>
Co-authored-by: Nikhil <nikhil.n.gohil@gmail.com>
Co-authored-by: ApolloVulpez <ambientskai@gmail.com>
Co-authored-by: skiinganchor <skiing_anchor.k4gaf@slmails.com>
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Giacomo Sfratato <xbit18@hotmail.it>